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DETAILED ACTION 
Response to Amendment 

1 . The amendment filed on 8/24/2006 has been entered. 



Response to Arguments 

2. The arguments filed on 8/24/2006 have been fully considered, however, they are 
deemed not to be persuasive. Kitamura teaches to one of ordinary skill in the art at the 
time of applicants invention determining the interaction constraints of the real object 
prior to controlling the position and orientation of the virtual object. Kitamura 
additionally teaches to one of ordinary skill in the art at the time of applicants invention 
the user both determines the constraints of the real object and controls the position and 
orientation of the virtual object. See page 134 first paragraph of section 2, page 135 
first paragraph of section 5, as well as the discussion given in sections 5.1-6. The 
discussion at page 135 section 5 "The user employs a six degrees of freedom (DOF) 
tracker device to manipulate virtual objects. The shapes of the real objects are 
assumed to be known in advance, and the motion of the real objects is also obtained by 
the 6 DOF tracker." teaches to determine first the constraints of the real object and then 
to control the position and orientation of the virtual object. The article as a whole 
teaches the user may perform both tasks of determining first the constraints of the real 
object and inputting instructions controlling the position and orientation of the virtual 
object with regard to real objects. The article also teaches users Kitamura and Kishino 
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determined for the toy block example, sections 4.1-6, the real objects 1 constraints and 
manipulated the position and orientation of the virtual object. 

The claims only generally claim the user inputs three dimensional information of 
a plurality of positions in real space and do not claim any specific device that is different 
from that which is used by Kitamura and Kishino. Thus, the claims need to be amended 
to claim a user controlled device that inputs three dimensional information of a plurality 
of positions in real space that is different than Kitamura and Kishino's user controlled 
device that inputs three dimensional information of a plurality of positions in real space, 
however, such a claimed difference will need to be analyzed under 35 (JSC 102 and 
103 in view of discussion at page 135 section 5 where the same 6 DOF input device for 
controlling the movement of the virtual object is used to control movement of the real 
object since movement of a real object would include movement of the real object with 
regard to a virtual object based upon the real object's constraints and in view of the 
position input devices listed on page 134 in section 2. 

The previous rejection is reproduced below with changes to reflect the 
amendments made to claims 10 and 16. 

Claim Rejections - 35 USC § 102 

3. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 1 02 that 
form the basis for the rejections under this section made in this Office action: 
A person shall be entitled to a patent unless - 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public 
use or on sale in this country, more than one year prior to the date of application for patent in the United 
States. 
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4. Claims 1-3 and 5-18 are rejected under 35 U.S.C. 102(b) as being anticipated by 
Yoshifumi Kitamura and Fumio Kishino, Consolidated Manipulation of Virtual and Real 
Objects, September 1997, Proceedings of the ACM symposium on Virtual reality 
software and technology, pages 133-138. Kitamura teaches an augmented reality 
system that uses object constraints to control the visual interaction between the virtual 
objects and the real objects. Claim 10 does not have an image taking step and thus 
does not claim the obtaining a constraining shape is separate from the image taking 
step. Therefore Kitamura teaches this broad method. 

A detailed analysis of the claims follows. 

Claim 10: 

Kitamura teaches an information processing method for changing the position 
and orientation of a virtual object in mixed reality space obtained by combining a real 
image and a virtual image, said method comprising the steps of: 

obtaining a constraining shape from a plurality of positions in real space (The 
measurements by the user in three dimensional real space, the device for obtaining the 
3-D shape by using a range image, and the device for obtaining the 3-D shape by using 
multiple cameras.) designated by a user (The user measures, the user directs the range 
measuring device to locations in the real world, and the user directs the cameras to 
locations in the real world.) using an operating unit (Ruler or range device or multiple 
camera device.) capable of obtaining three-dimensional positional information (Each of 
the devices, ruler, range and multiple camera obtain three-dimensional information.)". 
(The three paragraphs found in section 2 on page 134 of Kitamura teaches the user 
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measuring the real objects, a device measuring the real objects with range in response 
to the user, or a device measuring the real objects with multiple cameras range in 
response to the user. Kitamura discusses in section 2 in the first paragraph using 
conventional modeling software after precisely measuring the size or length of the real 
object by hand or by the devices which is an operating unit capable of obtaining three- 
dimensional positional information. ); 

changing the position and orientation of the virtual object according to 
instructions from the user, based on the obtained constraining shape as constraint 
condition (The first paragraph in section 5 on page 135 describes the user using a 6 
DOF tracker device to manipulate the virtual objects. The shape of the real objects are 
used to constrain the movement of the virtual object by giving the real object a shape 
that the virtual object interacts with in a constrained manner. Sections 5. 1 to 5.4 
discusses manipulation of the virtual object based upon constraint conditions based on 
the shape of the real object. The shape generated from the external instructions 
constraints the interaction of the virtual object with the real object, see sections 5. 1 to 
5.4.)\ and 

combining an image of the virtual object generated according to the changed 
position and orientation, and the real image (The introduction on page 133 second full 
paragraph discusses augmented reality which synthesizes a virtual object with a real object. 
The claim does not limit the form of the real image, thus, Kitamura teaches this limitation. 
Section 5. 1 discusses after movement is detected by the 6DOF manipulator the virtual object 
is moved according to the constraints.). 
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Claim 11: 

Kitamura teaches an information processing method according to Claim 10, 
further comprising the step of combining a virtual image indicating the constraining 
shape with the real image (The introduction on page 133 second full paragraph discusses 
augmented reality which synthesizes a virtual object with a real object. The claim does not 
limit the form of the real image, thus, Kitamura teaches this limitation. Section 5. 1 discusses 
after movement is detected by the 6DOF manipulator the virtual object is moved according to 
the constraints.). 

Claim 12: 

Kitamura teaches an information processing method according to Claim 10, 
wherein the constraining shape is a plane. (On page 136 in the text above figure 2 determining a 
plane and using the plane to constrain movement of the virtual object b discussed with regards to figure 
2). 

Claim 13: 

Kitamura teaches an information processing method according to Claim 10, 
wherein said changing step changing the position and orientation of the virtual object is 
carried out by changing the position and orientation of the operating unit (The 6 DOF 
tracker is an operating unit. The user using the 6 DOF tracker device manipulates the 
virtual objects.). 

Claim 14: 

Kitamura teaches a computer program product comprising a computer readable 
medium storing computer program code for performing the information processing 



Application/Control Number: 10/615,941 Page 7 

Art Unit: 2628 

method according to Claim 10 wherein the information processing method is executed 
by a computer device (This article is directed to computers that generate the augmented 
reality scene since it was published by ACM for a symposium on virtual reality software 
and technology and since at page 133 in the last sentence in the second paragraph of 
section 1 software/hardware is discussed. Software causing a computer to perform 
Kitamura's augmented reality is inherently stored in a computer readable recording 
medium.). 

Claim 15: 

Kitamura teaches a computer-readable recording medium, storing the computer 
program according to Claim 14 (Software causing a computer to perform Kitamura's 
augmented reality is inherently stored in a computer readable recording medium.). 

Claim 16: 

The functions of Kitamura corresponds to the claimed units because the software 
causes the computer to become a unit that performs a process. As seen below 
Kitamura performs the claimed processes. 

Kitamura teaches an information processing device for aiding control operations 
relating to controlling the position and orientation of a virtual object, said device 
comprising: 

an image capturing unit configured to capture a real image in real space (The 
measurements by the user in three dimensional real space, the device for obtaining the 
3-D shape by using a range image, and the device for obtaining the 3-D shape by using 
multiple cameras captures a real image in real space. The three paragraphs found in 
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section 2 on page 134 ofKitamura teaches the user measuring the real objects, a device 
measuring the real objects with range in response to the user, or a device measuring the 
real objects with multiple cameras range in response to the user. The claim does not limit 
the form of the real image, thus, Kitamura teaches this limitation.); 

a virtual image generation unit configured to generate a virtual image of a virtual 
object according to the position and orientation of said image capturing unit {Kitamura 
discusses in section 2 in the first paragraph using conventional modeling software after 
precisely measuring the size or length of the real object by hand or by the devices which 
is an generation unit capable of generating three-dimensional positional information.) ; 

a superimposing unit configured to superimpose the generated virtual image with 
the captured real image (Section 2 discusses mixing the virtual and real world objects.); 

an inputting unit configured to input three-dimensional position information of a 
plurality of positions inputted by a user in the real space (The measurements by the 
user in three dimensional real space, the device for obtaining the 3-D shape by using a 
range image, and the device for obtaining the 3-D shape by using multiple cameras are 
used to configure the constraints on the real world objects by using the three 
dimensional coordinates of the real world shape. ); 

a setting unit configured to set a constraining shape based on the inputting three- 
dimensional position information (The first paragraph in section 5 on page 135 describes 
the user using a 6 DOF tracker device to manipulate the virtual objects. The shape of the 
real objects are used to constrain the movement of the virtual object by giving the real 
object a shape that the virtual object interacts with in a constrained manner.) ; and 
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an operating unit configured to control the position and orientation of the virtual 
object based on the constraining shape in accordance with the a user's instruction^ 
Sections 5. 1 to 5.4 discusses manipulation of the virtual object based upon constraint 
conditions based on the shape of the real object The shape generated from the external 
instructions constrains the interaction of the virtual object with the real object, see 
sections 5. 1 to 5.4. The introduction on page 133 second full paragraph discusses 
augmented reality which synthesizes a virtual object with a real object. Section 5. 1 discusses 
after movement is detected by the 6DOF manipulator the virtual object is moved according to 
the constraints.) 

Claim 17: 

Kitamura teaches an information processing device according to Claim 16, 
wherein the constraining shape is defined by polygons (See section 4.1 second 
paragraph. Polyhedria includes polygons.) and the apexes of the polygons are at 
positions inputted by the user (According to section 2 the input points are used to form 
the real objects in the virtual world, thus, the apexes of the polyhedria correspond to the 
position inputted by the user by use of the ruler, range device, or device using multiple 
cameras.) or the constraining shape is a plane passing through the positions inputted by 
the user (A real sensed surface is a planar real world object). 

Claim 18: 

Kitamura teaches an information processing device according to Claim 16, 
wherein said operating unit performs at least one of the following operations in 
performing an operation controlling the position and orientation of the virtual object: 
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a translation operation for causing translational movement of the virtual object 
based on the constraining shape (Page 137 column 1 lines 1-3 teaches translation 
when the virtual object collides with the real surface.)] or 

a rotation operation for rotating the virtual object on an axis which is a normal 
vector at a plane where the constraining shape and the virtual object come into contact 
(Page 137 column 1 lines 1-3 teaches rotation when the virtual object collides with the 
real surface.) (Page 137 column 1 lines 1-3 also teaches translation and rotation when 
the virtual object collides with the real surface). 

Claims 1-3: 

Means plus function claims 1-3 correspond to device claims 16-18 and the 
means of Kitamura, software and computer, are equivalent to applicant's means of 
software and computer. 

Claims 5-7: 

Step for claims 5-7 correspond to device claims 16-18 and the steps of Kitamura, 
software and computer, are equivalent to applicant's steps performed by software and 
computer. 

Claims 8 and 9: 

Claims 8 and 9 mirror claims 14 and 15 addressed above and they are rejected 
for the same reasons given above for claims 14 and 15. 
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Prior Art 

5. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. 

Sato et al., US Patent No. 7,084,887,teaches at column 7 lines 37-54 using 
obstacles as markers and prior to sensing the markers inputting the locations of the 
markers. 

6. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the mailing date of this final action. 

7. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Jeffery A Brier whose telephone number is (571 ) 272- 
7656. The examiner can normally be reached on M-F from 7:00 to 3:30. If attempts to 
reach the examiner by telephone are unsuccessful, the examiner's supervisor, Michael 
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Razavi, can be reached at (571) 272-7664. The fax phone Number for the organization 
where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). 




Jeffery A Brier 
Primary Examiner 
Division 2628 



